LP-NORMS, LOG-BARRIERS AND CRAMER TRANSFORM 

IN OPTIMIZATION 



J.B. LASSERRE AND E.S. ZERON 

Abstract. We show that the Laplace approximation of a supremum 
by L p -norms has interesting consequences in optimization. For instance, 
the logarithmic barrier functions (LBF) of a primal convex problem P 
and its dual P* appear naturally when using this simple approximation 
technique for the value function g of P or its Legendre-Fenchel conjugate 
g* . In addition, minimizing the LBF of the dual P* is just evaluating 
the Cramer transform of the Laplace approximation of g. Finally, this 
technique permits to sometimes define an explicit dual problem P* in 
cases when the Legendre-Fenchel conjugate g* cannot be derived explic- 
itly from its definition. 



1. Introduction 

Let / : X — > M and u) : X — > K m be a pair of continuous mappings defined 
on the convex cone X C W 1 . Consider the function g : M m — > M U {—00} 
given by the formula: 

(1.1) y h-> g(y) := sup {/(x) : w(x) < y, x G X}. 

X 

For each fixed y 6 M. m , computing g(y) is solving the optimization problem 

(1.2) P: SU p{/(x): W (x)<y,xeX}, 

X 

and g is called the value function associated with P. The value function g 
provides a systematic way to generate a dual problem P* via its Legendre- 
Fenchel conjugate denoted g* : M. m — > R U {—00}. In the concave version 
(i.e. when g and g* are concave instead of convex), the Legendre-Fenchel 
conjugate g* is defined by 

(1.3) X^g*(X) := inf {X'y-g(y)}, 



and is finite on some domain D C K+. Then a dual problem is defined by: 
(1.4) P* : g(y) := (g*)*(y) = inf {A'y - g*(X)}. 

Of course, one has the property g(y) > g(y) because from 
g*(X) = inf{A'x - 5 (x) } < X'y-g(y), 
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one may deduce 

9(y) = (<7*)*(y) = inf{A'y- <7*(A)} > inf{A'y + 5 (y)-A'y} = gfr). 

A A 

Moreover, notice that: 

g(y) = inf {A'y - g* (A)} = inf {A'y + sup{#(z) - A'z} } 

A A z 

= inf {A'y + sup {sup {/(x) : w(x) < z} — A'z} } 
A z xex 

r +00 ifA^Mf 1 

= { inf {A'y + sup {/(x) — A'w(x)} } otherwise 
[ AeM™ xeX 

(1.5) = inf sup{/(x) + A(y- W (x)}. 

Finally, noting that inf {/(x) + A'(y — w(x) } = /(x) if w(x) < y and —00 
otherwise, one may write 

(1.6) g(y) = sup ^inf, {/(x) + A'(y - w(x) } 

xex 



(1.7) < 5(y) = inf sup {/(x) + A'(y - w( x ) } [by (US])], 

AGK_|_ x gx 

and the equality g(y) = g(y) holds true under some convexity assumption. 

However, in general g* cannot be obtained explicitly from its definition 
(II. 3p . and for dual methods to solve P, the inner maximization in (jl.7p 
must be done numerically for each fixed A. A notable exception is the conic 
optimization problem where / and u are both linear mappings, for which 
the dual (jl.4[) has an explicit form in terms of A. Of course, alternative 
explicit duals have been proposed but they involve both primal (x) and 
dual (A) variables. In particular, the Wolfe [14] and Mond-Weir [TT] duals 
even allow to consider weakened notions of convexity like e.g. pseudo- or 
quasi-convexity. For a nice exposition and related references on this topic, 
the interested reader is referred to Mond |12] and the references therein. 

Contribution. Our contribution is to show that the simple and well- 
known Laplace approximation of a supremum via a converging sequence of 
L p -norms has interesting consequences in optimization, for both primal and 
dual problems P and P*. 

Recall that the celebrated Logarithmic Barrier Function (LBF in short) 
associated with a convex optimization problem P as in (|1.2p , or with its dual 
P* in (|1.4p when g* is explicit, is an important tool in convex optimization 
because of its remarkable mathematical properties. For instance, when the 
LBF has the self-concordanc$\ property then the associated Logarithmic 



function ip : T> — > R is called jt-self-concordant on T> C R™, k > 0, if ip is three times 
continuously differentiable in T>, and for all x £ T> and h £ R n , one has 

\V 3 ip(yi)[h,h,h]\ < 2k (h'V 2 tp(x)h) 3/2 , 

where V 3 ^(x)[ft, h, h] is the third differential of (p at x and h; see e.g. [6] p. 52]. 
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Barrier algorithm to solve P or its dual P*, runs in time polynomial in the 
input size of the problem; see e.g. [5] and [BJ p. 13, 51 and 60]. But the LBF 
is only one particular choice among many other interior penalty functions! 
Our main contribution is to provide a rationale behind the LBF as we show 
that the LBF can be obtained by approximating the "max" of a function 
over a domain by standard L p -norms on the same domain. The scalar 1/p 
becomes the parameter of the LBF and nice convergence properties hold 
when p — >• oo. More precisely: 

• We first show that the LBF (with parameter p) associated with the 
primal problem P appears naturally by using the simple and well-known 
Laplace approximation of a supremum via L p -norms, applied to the inner 
infimum in (jl.6p . It is a bit suprising to obtain an efficient method in this 
way. Indeed, the inner infimum in (|1.6p (which is exactly equal to /(x) 
when x is an admissible solution of P) is replaced with its "naive" Laplace 
approximation by L p -norms, and to the best of our knowledge, the efficiency 
of this approximation has not been proved or even tested numerically! 

• Similarly, when using the same Laplace L p -norm approximation tech- 
nique for the infimum in the definition (jl.3p of the conjugate function g*, we 
obtain a function <p p : R m — > M which: (a) depends on an integer parameter 
p and (b), is valid on the relative interior ri"D of some domain T> C M m . 
Theorem U] states that the minimum of <j> p converges to the minimum of P 
as p — > oo. In doing so for conic optimization problems, the set T> is just 
the feasible set of the (known) explicit dual problem P*, and (fi p is (up to a 
constant) the LBF with parameter p, associated with P*. So again, for conic 
programs, the simple Laplace approximation of a supremum by L p -norms 
permits to retrieve the LBF of the dual problem P*! Interestingly, Theorem 
[5] states that the function y i— > min^ ^> P (A; y) is nothing less than the Cramer 
transform of the Laplace approximation ||e* [|jy>($v y )) i where fi(y) C X is the 
feasible set of problem P and || • \\u> is the usual norm associated with the 
Lebesgue space LP . To the best of our knowledge, this interpretation of the 
Logarithmic Barrier algorithm (with parameter 1/p) for the dual P*, is new 
(although in the particular context of Linear Programming, this result was 
already alluded to in [8]). 

Analogies between the Laplace and Fenchel transforms via exponentials 
and logarithms in the Cramer transform have been already explored in other 
contexts, in order to establish nice parallels between optimization and proba- 
bility via a change of algebra; see e.g. Bacelli et al. pQ, Maslov |10| . Lasserre 
[9], and the many references therein. In probability, the Cramer transform 
of a probability measure has also been used to provide exact asymptotics of 
some integrals as well as to derive large deviation principles. For a nice sur- 
vey on this topic the interested reader is referred to Piterbarg and Falatov 

• Finally, an interesting feature of this Laplace approximation technique 
is to provide us with a systematic way to obtain a dual problem (jl.4p in cases 
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when g* cannot be obtained explicitly from its definition (|1.3p . Namely, in 
a number of cases and in contrast with g* , the function <p p (\;y) obtained by 
using the Laplace approximation of the conjugate function g* by L p -norms, 
can be computed in closed-form explicitly. Examples of such situations are 
briefly discussed. In the general case, 4> p is of the form hi(X;y) + /i2(A;p) 
where: for every A G riD fixed, /i2(A;p) — > as p — > oo, and for each fixed 
p, the function A i— > h,2(\;p) is a barrier for the domain D. This yields to 
consider the optimization problem 

P* : min{/ii(A;y) : A G V} 
A 

as a natural dual of P, and for which (fi p is an associated barrier function 
with parameter p. If g* is concave then strong duality holds. 

2. Main result 

We need some intermediary helpful results before stating our main result. 

2.1. Some preliminary results. Let L q (X.) be the usual Lebesgue space 
of integrable functions defined on a Borel- measurable set X C W 1 , and 
IHIl<?(X) ( or sometimes \\h\\ g ) be the associated norm 

IHIw(X) = \\h\\ q := M\h(x)\«<to\ ^ . 

To make the paper self-contained we prove the following known result. 

Lemma 1. Let X C R™ be any Borel-measurable set, and h G L q (X.) for 
some given q > 1, so that H^Hl^x) < 00 • Then: 

lim ||/i||lp( X ) = IHloo := ess sup |/i(x)|. 
p^oo xeX 

Proof. Notice that X may be an unbounded set. Suppose that \\h\\ q < oo 
for some given q > 1, and define A to be the essential suppremum of \h\ in 
X. The result is trivial when A = 0, so we assume that A G (0, oo). Then 

esssup/i(x) = A = lim (||/i/A|| L9(X )) 9/:P A 
xex p^°° 



lim 

p— >oo 



X 



1/p 



(2.1) > lim 



lim | jypc). 

p— >oo v ' 



A 

\h(x)\ p dx 

x 

It is also obvious that A > lim p \\h\\ p when A = oo. On the other hand, 
suppose that the essential suppremum A of \h\ in X is finite. Given an 
arbitrary parameter e > 0, there exists a bounded subset B C X with 
positive finite Lebesgue measure X(B) G (0, oo) such that |/t(x)| > A— e for 
every x G B. Then 

lim \\h\\ LP(x) > lim \\h\\ LP(B) > lim A(B) 1 ^(A-e) = A - e. 

p— >0O v ' p— i>CXD v ' p->00 
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Therefore, since e is arbitrary, combining the previous identity with (|2.ip 
yields the desired result lim p _»oo ||/i||x,p(x) = A. In the same way, assume that 
the essential supremum of \h\ in X is infinite. Given an arbitrary natural 
number N G N, there exists a bounded subset B C X with positive finite 
Lebesgue measure X(B) G (0,oo) such that |/i(x)| > N for every x G B. 
Then 

lim |H| L P (X ) > lim ||/i|| LP(B ) > lim X(B) 1 ^ P N = N. 

p— »oo 4 p— >oo % ' p— >oo 

Therefore, since iV is arbitrary, combining the previous identity with (|2.ip 
yields the desired result lim^oo ||/i||ip(x) = A = oo. □ 

Next we also need the following intermediate result. 

Lemma 2. For every p G N let U p C R n fre some open subset, and let 
hp : Up —> R be a sequence of functions indexed by the parameter p G N. 
Suppose that h p converges pointwise to a function h defined on an open 
subset U ofR n . Then: 

lim inf /i„(x) < inf /i(x), 

p->ooxet/p xe(7 

provided that the limit in the left side of the equation exists in the extended 
interval [—00,00). 

Proof. Suppose that the infinimum of h on U is equal to —00. For every 
N G K there is a point x G U such that h(x) < N, and so there is also 
an index po such that x G U p and h p (x) < N for every p > pq. Hence the 
infinimum of h p on U p is strictly less than N, and so 

lim inf h p (x) = —00 = inf /i(x), 

p->oo xef/p xGf/ 

because G M is arbitrary. On the other hand, assume that the infinimum 
of h on U is equal to A G R. For every e > there is a point x G ?7 such 
that /i(x) < A+e, and so there is also an index po such that x G £/ p and 
/ip(x) < A+e for every p > po- Since the infinimum of /i p on C/ p is strictly 
less than A+e and e > is arbitrary, 

lim inf h v (x) < A = inf h(x). 

p-s-ooxec/p ^ xgC/ 

□ 



2.2. L p -norm approximations for the primal. Let us go back to prob- 
lem P in (jl.ip where X C M. n is a convex cone, and let Z := Rip. Let 
X* C M n be the dual convex cone associated with X, and let x 1— > A(x) be 
the universal logarithmic barrier function associated with the convex cone 
X, that is, 

(2.2) x^A(x):=ln^y e~ x ' y ciy^ , xGintX, 
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where int X denotes the interior of X. See e.g. Giiller [5| and Giiler and 
Tuncel [5]. Next, let U C M n be the set 

(2.3) U : = {x G R™ : w(x) < y; x G intX}. 

Recalling that P is a maximization problem, the LBF associated with the 
(primal) problem P, and with parameter p G N, is the function ip p : ri — > R 
defined by: 

(2.4) x h+ V P (x) := /(x) + 1 ^-A(x) + ln(y - wfx)),- j . 

(In some references like e.g. [B], pV> p is rather used.) The LBF in convex pro- 
gramming dates back to Frisch [3] and became widely known later in Fiacco 
and McCormick [2]. For more details and a discussion, see e.g. den Hertog 
[BJ Chapter 2]. It is well-known that under some convexity assumptiongj, 
and if g(y) < oo, 

(2.5) g(y) = lim sup{^ P (x) : xG%}, 

p^oo x 

and the sequence of maximizers (x p ) p6 N drioiip v converges to a maximizer 
of P. 

We next provide a simple rationale that explains how the LBF naturally 
appears to solve problem P. 

Proposition 3. With H as in < fO|) . let y G W n andxeH. Then: 

= sup {A'(w(x)-y)-/i'x} 
(A,/i)eZxX* 

= lim ln||e A '(-W-y)^' x || LnZxX , ) 

p— >oo v ' 

(2.6) = lim - f A(x) - V ln(y - u;(x)), I 

p— >oo p \ * • / 

\ J =1 / 

Proof. The first equation is trivial whereas the second one follows from 
Lemma [TJ Next, 

l n || e A'(^(x)-y)-^'x|i x = lj n f f e pA'( W (x)-y)-M t , * dMdA 

P JzJx* 



-[A(px)-^ln(p(y- W (x))),] 

- m 

— [A(x) — ln(y — u(x))j — (m + n) lnp] 



For instance if the mappings — / and lu are convex twice continuous differentiable, the 
interior of the feasible set is bounded, and the Hessian —\/ 2 ip p is positive definite on its 
domain; see Den Hertog [6] page 2]. As p varies, the unique maximizer x(p) of lp p , called 
the p- center, lies on the so-called central path. 
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where we have used that for each p£N, A(px) = A(x) — nlnp because X* 
is a cone. □ 

Observe that 

■ r r,/ > , ,// / X n , / -i f /(x) if x G X and w(x) < y 
mf {/x + A y-wx ^ • 

(Mezxx* J V ; W K JJ ' J \ -oo otherwise, 

and so (jl.6p can be rewritten 

(2.7) 5 (y) = sup inf {/(x) + A'(y - W (x)) + A}, 

xew (A,/i)eZxX« 

Using Proposition [3] and lim p _ 5 . 00 (m + n) lnp/p = 0, yields 



(2.8) g(y) = sup { 



/(x) + lim - f - A(x) + V ln(y - w (x))j | I 



A direct application of Lemma [2] to ()2.8p yields 



/(x) + I -A(x) + ln(y - w (x)), j J 



g{y) < hrn sup < 

= lim sup {*0 p (x) : x € Ti}, 

p->co x 

and in fact, (12.51) states that one also has the reverse inequality, 

In other words, the LBF ip p appears naturally when one approximates 

„ , in i v {A'(y-w(x)) + /i'x}, 
(A,/i)eZxX« 

(whose value is exactly zero if w(x) < y and x G X), by the quantity 



I (-A(x)+J>(y-a;(x))A 



which comes from the Laplace approximation of a "sup" by L p -norms. 

For instance, in Linear programming where X = WL, x i— > c'x and x i— > 
w(x) = Ax for some vector c 6 R n and some matrix A G M. mxn , 

- I m n \ 

5(y) = lim sup {c'x + - V ln(y - Ax)j + V ln(x t ) : x G U }. 

2.3. P-norm approximations for the dual. We now use the same ap- 
proximation technique via L p -norms to either retrieve the known dual P* 
when it is explicit, or to provide an explicit dual problem P* in cases where 
g* cannot be obtained explicitly from its definition (11. 3p . Recall that if g 
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is concave, upper semi-continuous, and bounded from above by some linear 
function, then by Legendre-Fenchel duality, 



(2.9) 
(2.10) 



g(y) = inf {A'y -g*(X)} , where 

A 

g*(X) : = inf {X'y - g(y)} . 
y 



One can express g* in terms of the definition (II. ID of g and the involved 
continuous transformations / and uj. Namely, 



(2.11) 
(2.12) 



g*(X) = sup{ 5 (y)-A' y } 
y 

= sup sup {/(x)-A'y} 
y *ex, 

w(x)<y 



sup {/(x)-A'w(x)} if A > 0, 
xex 

+oo otherwise. 



Therefore the domain of definition T> C M m of g* is given by: 

(2.13) V : = (a E M m : A > 0, sup {/(x)-A'w(x)} < ool , 
I xex J 

with relative interior denoted by riP. Observe that D is convex because 
—5* is convex and proper on T>. 



Theorem 4. Let g and g* be as in U.l\) and \2. 10\) . respectively. Assume 
that g is concave, upper semi- continuous, and bounded from above by some 
linear function. Suppose that the relative interior x\T> is not empty and for 
every A 6 x\T> there exists an exponent q 3> 1 such that 



(2.14) 
Then: 



1 /(x)-A'o>(x) 



< 00. 



L9(X) 



1 /(x)-A'a ) (x) 



(2.15) ^(y) = lim inf i A'y + In 

p^oo XeriV [ 

Proof. In view of (12111) 

sup sup {e~ A ' y+/ W) 

y x6X I. J 



LP(X) 



E 



ln(pA-,) 



(2.16) 



5*(A) 



»W<y 

+OO 



if A G V 
otherwise. 
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Hypothesis (|2.14p and Lemma [T] allow us to replace the supremum in 
f|2.16j) by the limit of the L p -norms as p — > oo. Namely, 



9* (A) 



lim In 

p— >oo 



lim 

p— >oo 



[ [ e- pX ' y+pf ^ dy dx ) 

ixeX iu(x)<y J 

| In ( jf e - P A^(x) +P /(x) ^ 1/P _ £ M^j) 



(2.17) 



lim < In 

p— >oo 



J(x)-A'o;(x) 



i=i 



Hence from ()2.17p . equation (|2.9p can be rewritten as follows: 



5 (y) = Jrf^A'y-^A)} 





= inf lim < 


A'y + In 


e /(x)-A'w(x) 




A€ri£> p->-oo 




(2.18) 


> lim inf < 


A'y + In 


e /(x)-A' W (x) 


p^oo AGriX> 







LP(X) B 



E 



ln(pAj) 



where we have applied Lemma [2] in order to interchange the "inf" and "lim" 
operators. Notice that the terms between the brackets are the functions 
hp(X) of Lemma [2j On the other hand, given y £ M. m , let 

9(y) := {(x,z) € X x R™ : w (x) + z < y} C R n+m , 

so that whenever A € riP, 

" 3 /(x)+A'(y-u,(x)-z] 



e /(x) 


< 




LP(6(y)) 



LP(0(y)) 



< e x 'y 



J(x)-A'w(x)-A'z 



Ilp(XxR™) 



J(x)-A'o;(x) 



TT(A-n 1/p - 



By hypothesis (I2.14h . given A € riD fixed, the L p -norm term in the last 
above identity is finite for some p large enough. Therefore, by definition 
(II. ip . Lemma [H and the continuity of the logarithm, one obtains 



g(y) = ln{ sup e^ x ) } = lim In 
(x,z)ee(y) 



< lim inf < A'y + In 

p^ooAgriD 



J(x)-A'^(x) 



\ LP (B(y)) 



ln(pAj 



i=i 

which combined with (|2.18p yields the desired result (|2.15p . 



□ 
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Remark 1. Condition (j2.14j) can be easily checked in particular cases. For 
instance let x i— > w(x) := Ax for some matrix A G R mxn . 

• If X := W\. and x i->- /(x) := c'x + lnx d for some c and d in R n 

with d > 0. The notation x d stands for the monomial Hfc=i x fe k • 
Then / is concave and 

e P/(x)-pA'w(x)^ x _ / e p(c-A'A)'x x pd ( ^ x 



fr r(i + P d fc ) 

l\ (A' k x - Ck y^ k < °°> 

whenever A G riD := {A G R m : A > 0, A'A > c} and p G N. 
• If X = R m and x h-> /(x) := — x'Qx + c'x for some eel" and a 
symmetric (strictly) positive definite matrix Q G R nxn , then 

f e p/(x) - pVw(x W = f e - px ' Qx e p(c - A ' A) ' x (ix < oo, 

whenever A G ri"D := {A G X : A > 0} and p G N. 
Consider next the following functions for every p G N : 

r- ' 1 P 



(2.19) A i ^ P (A; y) := A'y + In ef^~ x ' u ^ - V 

LP(X) 



defined on some domain of R m , and 

(2.20) y^<7 P (y) :=inf {^(A;y) : AGriP} 

defined on R m . Recall that the Cramer transform (denoted C) applied to an 
integrable function u : R m — > R, is the Legendre-Fenchel transform (denoted 
F) of the logarithm of the Laplace transform (denoted C) of u, i.e., 

u*->C(u) = T olno £ (u). 

The Cramer transform is natural in the sense that the logarithm of the 
Laplace transform is always a convex function. For our purpose, we will 
consider the concave version of the Fenchel transform 

(2.21) u ' y [T(u)](X) = inf{A'y + £(y)}, 

y 

for u : R m — > R convex, so that —u is concave. We claim that: 

Theorem 5. The function y h-» pg p (y) defined in \2. 20\) is the Cramer 
transform of the function 



(2.22) y^~9 P (y):= [ e^ x W 

JU(y) 

where fi(y) := {x G X : w(x) < y} C R n . 



v 

^ p (n(y)) 



LP-NORMS AND CRAMER TRANSFORM IN OPTIMIZATION 



11 



Proof. The result follows from the definition of the Cramer transform C. 

g p i-+C(g p ) := T olno L{g p ) 
y^C(~g p )(y) = mf {A'y + [mo£(^)](A)}. 



Hence 



[£(3 P )](pA) = / e- pX ' y ~g P (y)dy 

= I e-p y y\ [ 



3 P/ «dx 



'xeX,w(x)<y 



dy 



xex 



dy 



y>w(x) 



3 p/(x)-pA'^(x) dx 



xex 

/(x)-A'w(x) 



m ^ 



m ^ 

p w— 

Lp(X) f - 1 - pAo 
7 = 1 J 



Therefore, 



[lno£(^ p )](pA) = In 



J(x)-A'w(x) 



LP(X) 



On the other hand, recall the definition of g p {y) given in (|2.20p - (|2,19l) . 



9 P (y) = , in f i A'y + hi 



J(x)-A'a,(x) 



ln(pAj 



Thus, with T as in (I2.2ip and V p := {z : pz £ V}, we obtain the desired 
result : 



P 9 P (y ) = x mf <j pX'y + In 



3 /(x)-A'a ) (x) 



Lp(X) 



X>(p**) 



Ae 



inf {pA'y+[lno£(5 p )](pA)) 
= inf {z'y+[lno£fe)]( Z )l 
= [J-ohio£(s p )](y) = [C(&)](y), 

For linear programming, this result was already obtained in [HJ |9]. 



□ 



Example 1. (Linear Programming) In this case set the cone X = and 
the functions /(x) := c'x and w(x) = Ax for some vector c £ M. n and matrix 
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Aei mx ". We easily have that 



e /(x)-A' W (x) 


" =f 







PCk 



for every p £ N and each A in the relative interior ri T> of the set 
(2.23) V = {A £ M. m : A'A > c, A > 0}. 

Hence from (pTL9j) 



1 



p (A;y) = A'y + -ln 



P 



3 /(x)-A'a ) (x) 



E 



ln(pAj 



ln(A' fc A - c fc ) 



E 



ln(Aj) m+n 



P 



P 



hip, 



One easily recognizes (up to the constant (m+n) [In p]/p) the LBF with 
parameter p, of the dual problem: 

P* : min {A'y : A'A > c, A > 0}. 

A 

Example 2. (The general conic problem) Consider the conic optimization 
problem 

min {c x : ux < y, x £ X}, 



for some convex cone X C M. n , some vector c £ R n , and some linear mapping 
uj : W 1 — > R m with adjoint mapping u* : R m -> R n . We easily have that 



gC'x— X'w x 


V 


e (c-w*A)'x 


V 




LP(X) 




LP(X) 



= [ eP^-^^'^dx = p- n ( e^-^^dx, 
Jx ix 

because X is a cone. Claim (|2.19|) reads 

ln(j>Aj 



p (A;y) = A'y+-ln 
p 



c'x—X'w x 



LP(X) 



E 



P 



(2.24) 



A'y + 



V>(o;*A-c) 



E 

3=1 



In Aj m+n 



P 



P 



hip, 



where ip : R n — > R is the universal LBF (|2.2j) associated with the dual cone 
X*, and with domain riP, where 

(2.25) V = {A £ R m : w*A-c £ X*, A > 0}. 

In (ftp (and up to a constant), one easily recognizes the LBF with parameter 
p, of the dual problem: 

P* : min {A'y : oj*X-c £ X*, A > 0}. 
A 
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Example 3. (Quadratic programming: non conic formulation) Consider 
symmetric positive semidefinite matrixes Qj G R nxn and vectors Cj G R n 
for j = 0, l,...,m. The notation Q >z (resp. Q y 0) stands for Q is 
positive semidefinite (resp. strictly positive definite). Let X := R n , /(x) := 
— x'Qox— 2cqX and let uj : W 1 — > M. m have entries Wj(x) := x'QjX+2c^x 
for every j = 1, . . . , m. For A G M. m with A > 0, define the real symmetric 
matrix Q\ G R nxn and vector cj £ I": 

m m 

Qa •= Qo + y~] AjQj and c A := c + ^ AjCj, 



so that 



J(x)-A'o;(x) 



3=1 
LP(X) 



3=1 



/ exp (— px'Q^x— 2pc' A x)(ix 
n/2 exp (pc^Q^Ca) 



det(pQ A ) 



< oo, 



whenever p G N and Qa 0. Therefore 



<^ P (A;y) 



A'y + - In 

p 



,ci-Au x 



P 

LP(X) 



£ 

3=1 



(2.26) 



A'y + c'aQ^ca 



E 

3=1 



lnA 3 



+ n 



In ( det Q A ) 
ln7r— lnp 



2p 



m- 



lnp 

P 



on the domain of definition riD := {A : A > 0, Qa >~ 0}. Again, in equation 
(|2.26p one easily recognizes (up to a constant) the LBF with parameter p, 
of the dual problem P*: 



mm 
A>0, Qa^O 



max < — x'Qox — 2c x — Xj (x'QjX+2cjX— y^) > 

3=1 

| A'y + max { - x'Q A x - 2c' A x)} | 



= mm 

A>0,Q A b0 

= min {A'y + c' A Q^c A : A > 0, Q A h 0} , 

A 

where we have used the fact that x* = Q A x ca G M. n is the unique optimal 
solution to the inner maximization problem in the second equation above. 

If — Qo >- and Qj >: 0, j = 1, . . . , m, then riP := {A : A > 0} because 
Qa >- whenever A > 0; in this case P is a convex optimization problem 
and there is no duality gap between P and P*. 
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2.4. An explicit dual. A minimization problem P* in the variables A G 
D C M m with cost function A i— > /i(A) is a natural dual of P in (11 .2f) if weafc 
duality holds, that is, if for every feasible solution A G 2? of P* and every 
feasible solution x G M™ of P, one has /(x) < h(X). Of course, a highly 
desirable feature is that strong duality holds, that is, the optimal values of 
P and P* coincide. 

In Examples CO El and El the function cft p defined in (12. 19ft can be decom- 
posed into a sum of the form: 

(2.27) X^<ft p (X;y) = h^X; y) + h 2 (X;p) 

where hi is independent of the parameter p. Moreover, if /i2(A;p) < oo 
for some A > fixed, the term h2(X;p) converges to zero when p — > oo; in 
addition, for fixed p, h 2 is a barrier as h2(X;p) — > 00 as A approaches the 
boundary of T>. One may also verify that 

h 1 (X) = X'y-g*(X), VAgD, 

where g* is the Legendre-Fenchel conjugate in (|2.10p . In fact, the above pre- 
vious decomposition is more general and can be deduced from some simple 
facts. Recall that problem P is given in fjl .2 j> . and 



y >-> 9(y) ■= sup {/(x) : w(x) < y, x G X}, 

X 

for a convex cone X C M n and continuous mappings / and u. Assume that g 
is concave, upper semi-continuous, and bounded from above by some affine 
function, so that Legendre-Fenchel duality yields 

g(y) = inf {A'y -g*(X)} , where 

A 

g*(X) := inf {X'y - g(y)} , 
y 

and where the domain of g* is the set T> C M m in (|2. 13|) . 

Lemma 6. Suppose that riD is not empty and for every X G riT> there 
exists an exponent g > 1 such that {2.1J$ holds. Then: 

(2.28) A •"►/»! (A; y) := lim p (A;y) = A'y-<7*(A), VA G rxT>. 

p— ^oo 

Proof. Let A G riD be fixed. The given hypothesis and Lemma [1] imply that 



lim In 

p— >oo 



e /(x)-A'uK*) 



= SU P{/( X )- A ' W ( X )} 



= sup sup {/(x)-A'y} = sup{c/(y)-A'y} = ~9*(A), 

y x£X, y 
w(x)<y 

and so ([2728|> follows from the definition (j2~T9l) of (ft p . □ 
As a consequence we obtain: 



LP-NORMS AND CRAMER TRANSFORM IN OPTIMIZATION 



15 



Corollary 7. Let V be as in $2.13\) with riV ^ 0, (j) p as in \2.19\) and let 
A i — y hi(X;y) be as in \2.28}) . Then the optimization problem 

(2.29) P* : min{/ii(A;y) : A G ri£>}. 

A 

is a dual of P. Moreover, if g is concave, upper- semicontinuous and bounded 
above by some affine function, then strong duality holds. 

Proof. By Lemma[6j /ii(A; y) = A'y - 5* (A) for all A G riP. And so if minP* 
(resp. maxP) denotes the optimal value of P* (resp. P), one has 

minP* = min{A'y — g*(X) : A G riD} > <7(y)=maxP, 

A 

where we have used that —g*(X) = sup z {<?(z) — A'z} > g(y) — A'y. Finally, 
if g is concave, upper semi-continuous and bounded above by some affine 
function, then — g* is convex. Therefore, as a convex function is continuous 
on its domain T> (which is convex) 

minP* = min{/ii(A;y) : AGriP} 

A 

= min{A'y -g*(X) : A € riD} 

= min{A'y-3*(A) : A G V] = g(y), 

A 

that is, strong duality holds. □ 

In a number of cases, the L p -norm approximation of g* can be obtained 
explicitly as a function of A, whereas g* itself cannot be obtained explicitly 
from (|1.3p . In this situation one obtains an explicit LBF (j) p with parameter 
p, for some dual P* of P, and sometimes an explicit dual problem P*. 
Indeed if (fi p is known explicitly, one may sometimes get its pointwise limit 
hi(X,y) in (|2.28j) . in closed form, and so P* is defined explicitly by f|2.29|) . 
With p fixed, computing </> p (A; y) reduces to compute the integral over a 
convex cone of an exponential of some function parametrized by A and p. 
Sometimes this can be done with the help of some known transforms like 
e.g. the Laplace or Weierstrass transforms, as illustrated below. 

Linear mappings and Laplace transform. Let oj : M. n — > R m be a linear 
mapping, with w(x) = Ax for some real matrix A G R mxn , and let X = TSQ. . 
Then 

ln || e /(x)-AMx)|| Lp(x) = g-tpA'Ayx^/lx)^ 

= iln (£[e p/ ](M'A)) . 

That is, the L p -norm approximation is the logarithm of the Laplace trans- 
form of the function e-^, evaluated at the point pA'X G M n . So if in problem 
P, the objective function / is such that e-^ has an explicit Laplace transform, 
then one obtains an explicit expression for the LBF A i— )■ (fi p (X; y) defined in 
(|2T9jh 
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For instance if /(x) = c'x + \nq(x) for some vector c G M n and some 
polynomial q G M[x], positive on the feasible set of P, write 

^( X ) P = 

a€N n 

for finitely many non zero coefficients (q pa ), and where the notation x Q stand 
for the monomial x" 1 • • • x" n . Then In (£[e p ^](pA'A)) can be computed in 
closed-form since we have: 

ln(£[e^]( P A'A)) = In ( ^ q m f e ^ c - A ' A )' x x a dx j , 

lnp+ln lS. ?p "^n'L 1 (A'A-c) i )' 



— nJ 



where = n£=i i^i- Of course the above expression can become quite 

i 

complicated, especially for large values of p. But it is explicit in the variables 
(Aj). If the function x i— > lnq(x) is concave, then Corollary [7] applies. On 
the other hand, to obtain g* explicity would require to solve A'A — c — 
Vq , (x)/g , (x) = in closed form, which is impossible in general. 

Similarly if / is linear, i.e. x i— > /(x) = c'x for some vector c G M ra , then 

l n || e /(x)-A'a;(x)|| Lp(x) = p -l ^ f^-pX'u,^ (pc) 



and so if the function x i— > e~P x/uJ ( x ) has an explicit Laplace transform then 
so does the L p -norm approximation, and again, (j) p is obtained in closed 
form. 

Example 4. As a simple illustrative example, consider the optimization 
problem: 

(2.30) P: sup J c'x + Vb fc lnx fc : Ax < y, x > 0, x G R n 1 , 
x I k=i J 

for some given matrix A G W mxn and vectors b,c G ffi n and y G M. m . We 
suppose that b > 0, so that c'x + ln(x b ) is concav^l, and in which case P 
is a convex program. Notice that with X = := {x G W l : x > 0}, 



sup | c'x + ln(x b ) - A' Ax : x G R n , x G X j < 
whenever A lies in V = {A G W n : A'A > c, A > 0}. 



'The notation x b stands for the monomial a;? 1 ■ ■ ■ x b n 
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Proposition 8. The function <j) p in \2. 19\) associated with the optimization 



problem 112.301) is given by: 

n 

(2.31) p (A;y) = A'y + £ 



lnr(l+pb fc ) 



k=l 

^ ln(A^,A-c fc 



V 



h k ln(pA' k X-pc k ) 



E 



P 



E" 1 In A 7 - m + n, 
— -i — In p. 



P 



k=i c j=i 
and is the LBF with parameter p, of the dual problem: 

~A' k \-c k 



P 



P* : inf<^ A'y-^bfcln 



k=l 



: A' A > c, A > 



In particular, strong duality holds, i.e., the optimal values o/P and P* are 
equal. 



Proof. We have 

c'x-A'Ax h 



X 



p 

LP(X) 



n 



3 p(c-A'A)'x xP b dx 

r(i+ P b fc ) 

L {pA' k \-pc k y+P^ 



< oo, 



whenever p£N and A > in M. m satisfies A' A > c (and where T is the usual 
Gamma function). Next, cp p in (I2.19P reads 



>(A;y) = A'y 



In 



V 



e c'x-A'Ax x b 



E 

k=l 



hx(jp\ k ) 



which is (|2.3ip . Next, Stirling's approximation T(l + t) ~ (t/e) t y/2irt for 
real numbers t > 1, yields 



lim — - — — — — h k hip = lim In 

p— >oo p p— >-oo 

= b fc ln(e _1 b fc ). 

By Lemma [U 

A'y - g* (A) = lim p (A; y) = A'y - b fc In 

p— >-oo * ' 



bfe lnp 



fe=i 



A^.A-Cfc 
e _1 b fe 



And so, by Corollary the function <p p is the LBF with parameter p, of 
the dual problem 



P*: inf \ A'y -^b* In 



fc=i 



A' fc A-c fc 
e _1 bz. 



A'A > c, A > 



In particular, strong duality holds. 



□ 
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